The Future of Collaborative Human-Artificial Intelligence Decision-Making for Mission Planning.

In an increasingly complex military operating environment, next generation wargaming platforms can reduce risk, decrease operating costs, and improve overall outcomes. Novel Artificial Intelligence (AI) enabled wargaming approaches, based on software platforms with multimodal interaction and visualization capacity, are essential to provide the decision-making flexibility and adaptability required to meet current and emerging realities of warfighting. We highlight three areas of development for future warfighter-machine interfaces: AI-directed decisional guidance, computationally informed decision-making, and realistic representations of decision spaces. Progress in these areas will enable development of effective human-AI collaborative decision-making, to meet the increasing scale and complexity of today's battlespace.

Autonomous Flying With Neuromorphic Sensing.

Autonomous flight for large aircraft appears to be within our reach. However, launching autonomous systems for everyday missions still requires an immense interdisciplinary research effort supported by pointed policies and funding. We believe that concerted endeavors in the fields of neuroscience, mathematics, sensor physics, robotics, and computer science are needed to address remaining crucial scientific challenges. In this paper, we argue for a bio-inspired approach to solve autonomous flying challenges, outline the frontier of sensing, data processing, and flight control within a neuromorphic paradigm, and chart directions of research needed to achieve operational capabilities comparable to those we observe in nature. One central problem of neuromorphic computing is learning. In biological systems, learning is achieved by adaptive and relativistic information acquisition characterized by near-continuous information retrieval with variable rates and sparsity. This results in both energy and computational resource savings being an inspiration for autonomous systems. We consider pertinent features of insect, bat and bird flight behavior as examples to address various vital aspects of autonomous flight. Insects exhibit sophisticated flight dynamics with comparatively reduced complexity of the brain. They represent excellent objects for the study of navigation and flight control. Bats and birds enable more complex models of attention and point to the importance of active sensing for conducting more complex missions. The implementation of neuromorphic paradigms for autonomous flight will require fundamental changes in both traditional hardware and software. We provide recommendations for sensor hardware and processing algorithm development to enable energy efficient and computationally effective flight control.

Abrupt darkening under high dynamic range (HDR) luminance invokes facilitation for high-contrast targets and grouping by luminance similarity.

When scanning across a scene, luminance can vary by up to 100,000-to-1 (high dynamic range, HDR), requiring multiple normalizing mechanisms spanning from the retina to the cortex to support visual acuity and recognition. Vision models based on standard dynamic range (SDR) luminance contrast ratios below 100-to-1 have limited ability to generalize to real-world scenes with HDR luminance. To characterize how orientation and luminance are linked in brain mechanisms for luminance normalization, we measured orientation discrimination of Gabor targets under HDR luminance dynamics. We report a novel phenomenon, that abrupt 10- to 100-fold darkening engages contextual facilitation, distorting the apparent orientation of a high-contrast central target. Surprisingly, facilitation was influenced by grouping by luminance similarity, as well as by the degree of luminance variability in the surround. These results challenge vision models based solely on activity normalization and raise new questions that will lead to models that perform better in real-world scenes.

A 100,000-to-1 high dynamic range (HDR) luminance display for investigating visual perception under real-world luminance dynamics.

BACKGROUND: Real-world illumination challenges both autonomous sensing and displays, because scene luminance can vary by up to 109-to-1, whereas vision models have limited ability to generalize beyond 100-to-1 luminance contrast. Brain mechanisms automatically normalize the visual input based on feature context, but they remain poorly understood because of the limitations of commercially available displays. NEW METHOD: Here, we describe procedures for setup, calibration, and precision check of an HDR display system, based on a JVC DLA-RS600U reference projector, with over 100,000-to-1 luminance dynamic range (636-0.006055 cd/m2), pseudo 11 bit grayscale precision, and 3 ms temporal precision in the MATLAB/Psychtoolbox software environment. The setup is synchronized with electroencephalography (EEG) and infrared eye-tracking measurements. RESULTS: We show display metrics including light scatter versus average display luminance (ADL), spatial uniformity, and spatial uniformity at high spatial frequency. We also show a luminance normalization phenomenon, contextual facilitation of a high contrast target, whose discovery required HDR display. COMPARISON WITH EXISTING METHODS: This system provides 100-fold greater dynamic range than standard 1000-to-1 contrast displays and increases the number of gray levels from 256 or 1024 (8 or 10 bits) to 2048 (pseudo 11 bits), enabling the study of mesopic-to-photopic vision, at the expense of spatial non-uniformities. CONCLUSIONS: This HDR research capability opens new questions of how visual perception is resilient to real-world luminance dynamics and will lead to improved visual modeling of dense urban and forest environments and of mixed indoor-outdoor environments such as cockpits and augmented reality. Our display metrics code can be found at https://github.com/USArmyResearchLab/ARL-Display-Metrics-and-Average-Display-Luminance.

EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces.

OBJECTIVE: Brain-computer interfaces (BCI) enable direct communication with a computer, using neural activity as the control signal. This neural signal is generally chosen from a variety of well-studied electroencephalogram (EEG) signals. For a given BCI paradigm, feature extractors and classifiers are tailored to the distinct characteristics of its expected EEG control signal, limiting its application to that specific signal. Convolutional neural networks (CNNs), which have been used in computer vision and speech recognition to perform automatic feature extraction and classification, have successfully been applied to EEG-based BCIs; however, they have mainly been applied to single BCI paradigms and thus it remains unclear how these architectures generalize to other paradigms. Here, we ask if we can design a single CNN architecture to accurately classify EEG signals from different BCI paradigms, while simultaneously being as compact as possible. APPROACH: In this work we introduce EEGNet, a compact convolutional neural network for EEG-based BCIs. We introduce the use of depthwise and separable convolutions to construct an EEG-specific model which encapsulates well-known EEG feature extraction concepts for BCI. We compare EEGNet, both for within-subject and cross-subject classification, to current state-of-the-art approaches across four BCI paradigms: P300 visual-evoked potentials, error-related negativity responses (ERN), movement-related cortical potentials (MRCP), and sensory motor rhythms (SMR). MAIN RESULTS: We show that EEGNet generalizes across paradigms better than, and achieves comparably high performance to, the reference algorithms when only limited training data is available across all tested paradigms. In addition, we demonstrate three different approaches to visualize the contents of a trained EEGNet model to enable interpretation of the learned features. SIGNIFICANCE: Our results suggest that EEGNet is robust enough to learn a wide variety of interpretable features over a range of BCI tasks. Our models can be found at: https://github.com/vlawhern/arl-eegmodels.

Network Anisotropy Trumps Noise for Efficient Object Coding in Macaque Inferior Temporal Cortex.

How neuronal ensembles compute information is actively studied in early visual cortex. Much less is known about how local ensembles function in inferior temporal (IT) cortex, the last stage of the ventral visual pathway that supports visual recognition. Previous reports suggested that nearby neurons carry information mostly independently, supporting efficient processing (Barlow, 1961). However, others postulate that noise covariation effects may depend on network anisotropy/homogeneity and on how the covariation relates to representation. Do slow trial-by-trial noise covariations increase or decrease IT's object coding capability, how does encoding capability relate to correlational structure (i.e., the spatial pattern of signal and noise redundancy/homogeneity across neurons), and does knowledge of correlational structure matter for decoding? We recorded simultaneously from ∼80 spiking neurons in ∼1 mm(3) of macaque IT under light neurolept anesthesia. Noise correlations were stronger for neurons with correlated tuning, and noise covariations reduced object encoding capability, including generalization across object pose and illumination. Knowledge of noise covariations did not lead to better decoding performance. However, knowledge of anisotropy/homogeneity improved encoding and decoding efficiency by reducing the number of neurons needed to reach a given performance level. Such correlated neurons were found mostly in supragranular and infragranular layers, supporting theories that link recurrent circuitry to manifold representation. These results suggest that redundancy benefits manifold learning of complex high-dimensional information and that subsets of neurons may be more immune to noise covariation than others. SIGNIFICANCE STATEMENT: How noise affects neuronal population coding is poorly understood. By sampling densely from local populations supporting visual object recognition, we show that recurrent circuitry supports useful representations and that subsets of neurons may be more immune to noise covariation than others.

Corrigendum: Correlated activity supports efficient cortical processing.

[This corrects the article on p. 171 in vol. 8, PMID: 25610392.].

Tuning and spontaneous spike time synchrony share a common structure in macaque inferior temporal cortex.

Investigating the relationship between tuning and spike timing is necessary to understand how neuronal populations in anterior visual cortex process complex stimuli. Are tuning and spontaneous spike time synchrony linked by a common spatial structure (do some cells covary more strongly, even in the absence of visual stimulation?), and what is the object coding capability of this structure? Here, we recorded from spiking populations in macaque inferior temporal (IT) cortex under neurolept anesthesia. We report that, although most nearby IT neurons are weakly correlated, neurons with more similar tuning are also more synchronized during spontaneous activity. This link between tuning and synchrony was not simply due to cell separation distance. Instead, it expands on previous reports that neurons along an IT penetration are tuned to similar but slightly different features. This constraint on possible population firing rate patterns was consistent across stimulus sets, including animate vs. inanimate object categories. A classifier trained on this structure was able to generalize category "read-out" to untrained objects using only a few dimensions (a few patterns of site weightings per electrode array). We suggest that tuning and spike synchrony are linked by a common spatial structure that is highly efficient for object representation.

Tuning dissimilarity explains short distance decline of spontaneous spike correlation in macaque V1.

Fast spike correlation is a signature of neural ensemble activity thought to underlie perception, cognition, and action. To relate spike correlation to tuning and other factors, we focused on spontaneous activity because it is the common 'baseline' across studies that test different stimuli, and because variations in correlation strength are much larger across cell pairs than across stimuli. Is the probability of spike correlation between two neurons a graded function of lateral cortical separation, independent of functional tuning (e.g. orientation preferences)? Although previous studies found a steep decline in fast spike correlation with horizontal cortical distance, we hypothesized that, at short distances, this decline is better explained by a decline in receptive field tuning similarity. Here we measured macaque V1 tuning via parametric stimuli and spike-triggered analysis, and we developed a generalized linear model (GLM) to examine how different combinations of factors predict spontaneous spike correlation. Spike correlation was predicted by multiple factors including color, spatiotemporal receptive field, spatial frequency, phase and orientation but not ocular dominance beyond layer 4. Including these factors in the model mostly eliminated the contribution of cortical distance to fast spike correlation (up to our recording limit of 1.4mm), in terms of both 'correlation probability' (the incidence of pairs that have significant fast spike correlation) and 'correlation strength' (each pair's likelihood of fast spike correlation). We suggest that, at short distances and non-input layers, V1 fast spike correlation is determined more by tuning similarity than by cortical distance or ocular dominance.

Correlated activity supports efficient cortical processing.

Visual recognition is a computational challenge that is thought to occur via efficient coding. An important concept is sparseness, a measure of coding efficiency. The prevailing view is that sparseness supports efficiency by minimizing redundancy and correlations in spiking populations. Yet, we recently reported that "choristers", neurons that behave more similarly (have correlated stimulus preferences and spontaneous coincident spiking), carry more generalizable object information than uncorrelated neurons ("soloists") in macaque inferior temporal (IT) cortex. The rarity of choristers (as low as 6% of IT neurons) indicates that they were likely missed in previous studies. Here, we report that correlation strength is distinct from sparseness (choristers are not simply broadly tuned neurons), that choristers are located in non-granular output layers, and that correlated activity predicts human visual search efficiency. These counterintuitive results suggest that a redundant correlational structure supports efficient processing and behavior.

Orientation domain diversity in macaque area V2.

Single orientation domains in primary (V1) and second (V2) visual cortical areas are known to encode the orientation of visual contours. However, the visual world contains multiple and complex contour types. How do these domains handle such complexity? Using optical imaging methods, we have examined orientation response to two types of contours: real (luminance-defined) and illusory (inferred). We find that, unlike area V1, there are multiple types of orientation domain in V2. These include "real only" domains, "higher-order" domains (which respond to an orientation whether real or illusory), and other domains with nonmatching real/illusory orientation preference. We suggest that this plurality of orientation domains in V2 enables the complexities of local and global contour extraction.

Hippocampal desynchronization of functional connectivity prior to the onset of status epilepticus in pilocarpine-treated rats.

Status epilepticus (SE), a pro-epileptogenic brain insult in rodent models of temporal lobe epilepsy, is successfully induced by pilocarpine in some, but not all, rats. This study aimed to identify characteristic alterations within the hippocampal neural network prior to the onset of SE. Sixteen microwire electrodes were implanted into the left hippocampus of male Sprague-Dawley rats. After a 7-day recovery period, animal behavior, hippocampal neuronal ensemble activities, and local field potentials (LFP) were recorded before and after an intra-peritoneal injection of pilocarpine (350 mg/kg). The single-neuron firing, population neuronal correlation, and coincident firing between neurons were compared between SE (n = 9) and nonSE rats (n = 12). A significant decrease in the strength of functional connectivity prior to the onset of SE, as measured by changes in coincident spike timing between pairs of hippocampal neurons, was exclusively found in SE rats. However, single-neuron firing and LFP profiles did not show a significant difference between SE and nonSE rats. These results suggest that desynchronization in the functional circuitry of the hippocampus, likely associated with a change in synaptic strength, may serve as an electrophysiological marker prior to SE in pilocarpine-treated rats.

A functional circuitry for edge-induced brightness perception.

The identification of visual contours and surfaces is central to visual scene segmentation. One view of image construction argues that object contours are first identified and then surfaces are filled in. Although there are psychophysical and single-unit data to suggest that the filling-in view is correct, the underlying circuitry is unknown. Here we examine specific spike-timing relationships between border and surface responses in cat visual cortical areas 17 and 18. With both real and illusory (Cornsweet) brightness contrast stimuli, we found a border-to-surface shift in the relative timing of spike activity. This shift was absent when borders were absent and could be reversed with relocation of the stimulus border, indicating that the direction of information flow is highly dependent on stimulus conditions. Furthermore, this effect was seen predominantly in 17-18, and not 17-17, interactions. These results demonstrate a border-to-surface mechanism at early stages of visual processing and emphasize the importance of interareal circuitry in vision.

Object selectivity of local field potentials and spikes in the macaque inferior temporal cortex.

Local field potentials (LFPs) arise largely from dendritic activity over large brain regions and thus provide a measure of the input to and local processing within an area. We characterized LFPs and their relationship to spikes (multi and single unit) in monkey inferior temporal cortex (IT). LFP responses in IT to complex objects showed strong selectivity at 44% of the sites and tolerance to retinal position and size. The LFP preferences were poorly predicted by the spike preferences at the same site but were better explained by averaging spikes within approximately 3 mm. A comparison of separate sites suggests that selectivity is similar on a scale of approximately 800 microm for spikes and approximately 5 mm for LFPs. These observations imply that inputs to IT neurons convey selectivity for complex shapes and that such input may have an underlying organization spanning several millimeters.

Fast readout of object identity from macaque inferior temporal cortex.

Understanding the brain computations leading to object recognition requires quantitative characterization of the information represented in inferior temporal (IT) cortex. We used a biologically plausible, classifier-based readout technique to investigate the neural coding of selectivity and invariance at the IT population level. The activity of small neuronal populations (approximately 100 randomly selected cells) over very short time intervals (as small as 12.5 milliseconds) contained unexpectedly accurate and robust information about both object "identity" and "category." This information generalized over a range of object positions and scales, even for novel objects. Coarse information about position and scale could also be read out from the same population.

Cortical processing of a brightness illusion.

Several brightness illusions indicate that borders can affect the perception of surfaces dramatically. In the Cornsweet illusion, two equiluminant surfaces appear to be different in brightness because of the contrast border between them. Here, we report the existence of cells in monkey visual cortex that respond to such an "illusory" brightness. We find that luminance responsive cells are located in color-activated regions (cytochrome oxidase blobs and bridges) of primary visual cortex (V1), whereas Cornsweet responsive cells are found preferentially in the color-activated regions (thin stripes) of second visual area (V2). This colocalization of brightness and color processing within V1 and V2 suggests a segregation of contour and surface processing in early visual pathways and a hierarchy of brightness information processing from V1 to V2 in monkeys.

Weakly modulated spike trains: significance, precision, and correction for sample size.

Many single-unit electrophysiological studies of visual cortex have investigated strong evoked responses to simple stimuli such as oriented gratings. Experiments involving other types of stimuli, such as natural scenes, higher-order features, and surface brightness, produce single-unit responses that are more difficult to interpret. Experiments with brightness, in particular, evoke single-unit responses that are typically weakly modulated. When the brightness is generated by a visual illusion such as the Cornsweet illusion, statistical tests are often necessary to distinguish true responses from baseline fluctuations. Here, using data collected from cat Areas 17 and 18 in response to real and illusory brightness stimuli, we provide a method for detecting and quantifying weak but significant periodic responses. By randomizing spike trains (via bootstrap methods), we provide confidence levels for response significance, permitting the evaluation of both weak and strong responses. We show that because of a strong dependence on total spike number, response significance can only be appropriately determined with randomized spike trains of similar spike number. Such randomizations can be performed for both stimulus-elicited and spontaneously occurring spike trains. By developing a method for generating randomized modulated spike trains (phase-restricted randomization) from actual recordings, we calculate upper and lower confidence limits of modulated spike trains and describe how measurement precision varies as a function of total spike count. Finally, using this randomization method, we describe how a correction function can be determined to correct for measurement bias introduced at low spike counts. These methods may also be useful in the study of small but potentially significant responses in other systems.